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y ' PREFACE 



Amendments to TitlQ I of the Elementary and Secondary Act in 1974 
Included a requirement for the Commissioner of Education to develop ' 
evaluation models and standards for use by State and local educational 
agencies. Experience in using the models which have been developed is 
the main subject of this report. ^ . ' 



: Although the models have not beenTequtireSlsy^regirlation, all States 
have been considering the models and approximately "5500 the schoo] 
districts receiving Titl6 I funds aire now using them to some degree* All 
this activity advances us toward t*ie goal of better Title I evaluations 
but we are especially incfebt^ to, a. number of individuals for theii: extra-/- 
ordinary efforts in testing the models and sharing their experienceswith 
US/ They include Don Christie and Vinc^McInnls (Maine); Cliodt Berndt 
(South Dakota); Chuck Statler (South .Carolina); Arlie Cox and Carl Ev.ans ^ 
(Ohio); Oliver Himley and Ron Huff (Iowa); Mary Ann DeLong, (Ocala, Florida); 
Mel Lucas (Gainesville^ .Florida); and Perry Geiger (Bronsoh, FloHda)v 

'As the report reveals", use of the models in school- districts suggests 
many ways to either Improve the quality^of Title I evaluations oc ,to l^sen 
the^work involved*. These changes will^be m^ide as we, contiKtie to refine the 
iTitle I evaluation and reporttng system. ' / ' " 

' / ■ ' ' ^ ' ^ ^ . ' . \. 

\ ^ =^ Carl t/Wisler / 

/ , , I Director, Elementary and Secondary ^ 

/ . , A: Programs Div.ision 

/ 7 Office of Planning, Budgeting, 

— *' ^ , - \ . and Evaluation 



EXECUTIVE SUMMARY 
: ^ . ' . ^ - 

In August of 1974, Congress amended ESEA Title I adding to it Section 
151 which requires the Comnissioner of Edi/cation to develop uniform pro- 
cedures--models— for use by local and State personnel in their annuaV 
evaluations of Title I projects* Also required is a program of -technical 
assistance to. help them use the procedures* Work to develop the models 
emphasized (1) the validity and comparai)>lity of project evalijationa and 
(2) the flexibiVity necessary to make the procedures appropriate for use In 
most Title I sites^. Prototypes were reviewed by personnel in nine States 
during the spring of 197S, by program and eva^uAtlon staffs in alT States, 
(and HLOver 400 school districts) between June 1975 and June 197i6, and 
agaifK^by^all State personnel in national training workshops latei^ in T976. 
This review process was instrumental in soliciting input frpm persons who 
would be eventually using the models and in persuading many to use the 
evaluation models* before any formal requirement was issued. 

This is a sutnmary of an extensive report about the experi^ences of , ' * 
people using the models during the 1976-77 school year. Included .in the 
report are descriptions (1) of the stat^us oT rwtionwlde use of the models, 
(2) of State activities to change the evaluation practices they require of 
their locaf^istricts, (3) of local processes to collect, analyse, and re- 
port/dataya(ccqrding to the new procedures, (4) of progress, problems, and 
rec&nm^mfetions of those who are using the models, (5) of the .USOE/DHEW steps, 
to require use of the, methods by program regulations, and (6) of con^lusioni 
regarding the program and its direction in the near future. 

There were fifteen States in the 1976-77 school year with half or^ more 
of their school districts using the new standardized practices to evaluate 
their Title I projects. Those districts comprised about 25% of the total in 
■<We JJnited States receiving'Title I funds, Use of ithe models has grown^ con- 
siderably, and ibout 40% of the Title I districts are using them in thi^ year 
(1977^78). - ' ^ \ Y , 

The increase is probablj^ due to severaf factors including the substantial 
♦amount of help available to States(and at their d'iscretion, to local districts) 
from ten Technical* Assistance Centers established under contract to USOE V " 
■September of 1976, Their purpose is to offer advice* regarding site-specific 
evaluation problems in Title I, to assist States in changing to the new system 
(helping in the installation of new compt^ter programs, etc), anct to provioe' 
other technical support as needed by States in offering training workshops! - 
revising evaluation materials, etc* \ 

In the lafe fall of 1977, staff members from the Office of Plannjng, 
Budgeting, and Evaluation visited six of the fifteen States who had large 
rlumbers of districts using the, models in 1976-77 to discuss their experience^ 
withthe system. Based on those discussions they were able to describe the 
iV-step, 2-3 year process the SEA administrators had pursued to change evaluaV 
tion activities in, their locals to correspond more. closely with those advocated 
by USQE, The activities,* including the decision to'effect such a change. 
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id€(itificBittbn of pilot or""test" -sites, training of personnel, change in 
report^fonrats. monitoring of practifesi review of data, and re-iteration 
th^ subsequent year" to involve more sites have bee? labor intensive and 
exhausting {gk best; fn^trating'and probTan-riddled at worst). State 
personnel. havlp>jote'd some positive'outgrowths of the process, however, 
including greater attention to ..tests and theii" uses, more confidence in 
data used for \St3te-iden.tificatioVi of good pro3ects. and greater familiarity 
on the part of\ program personnel \With evaluation goal&ratid issues. 
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. To use the ^valuation sys^ent LEftI personnel must faithfully adhere to 
^?^ocedure^ outlined in 4;hree_^evaluation models concerning test selection dnd 
administration, jtest scoring,, score conversion, data analysis, and aggregation. 
Conv.ersations with staff members it both the State an^ local levels identified - 
both successes aVid difficulties they ^ncountered in using the techniques. 

Aspects of -tl^e 'system which they appl^uded included the clarity of 
instructions, the, thoroughness of ^SOE materials regarding the different 
approaches, .the production of data which-appeared more v^alid (with fewer results 
being counter-inti^itive; etc.). ' Remaining difficulties include njethods for 
selecting Title I students so as not to invalidate the evaluations' and the con- 
version of test scares to values along different scales. These i«ere iplentified 
by USOE and SEA stdff perusing 'LEA reports. Of the 183, examined , 32 had no 
discernai)le err^or which was viewed as evidence that the procedures, for the 
most part, are feasible and usable. Wor'k is underway bu USOE to address the 
remaining di.fficul tiles. . . ' \ 

^As local, StateVand Feder^rpersonnel encourage each other to use the 
uniform evaluation ^mqdels, tiiere are al so activities underway to require 
their-use.by Title'.I Vantees. These activities emphasize. public input 
thr^gh. discussion of\issue,s identified in a memorandum sent to all $tate 
superintendents, Title'I ?taffs. evaluators. and interested parties and 
through hearings aboutl draft rules (ttf be published next year. most. 1 ikely). ' 
Final regulatiofts might be in effkt soon enough, then, to ensure standard 
eva?uation activities dnd .repbrtihg far Title I projects the following , ' ■ . 

sctiopl year. Also underway a^e.some studies to determine when, enough data 
Wilrbe available whichlare sufficiently uniform and representative to support 
.nation,?!, statements abpiit the achievement of Title I students. . .■ 

Plajis .for tl^^ evaluation program at this.time and its directions in the 
near future', fficlude Continued support and'emphasis upon the services provided - 
by Tecjipical Assi stance£f nter staff, greater dissemination qf evaluation |p 
materials, the pursuit of\further technical documentation for the models, and . I 
conttnt|id*solicitatitin of Ipublic input. Another progress report regarding | 
these-'Ajrid other aspects of\;^e program will be available during the fall of 

iii 
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In sum, the Education Anjendments of J974 added Section 151 to Title I ^ 
of ESEA to make grantee evaluations of that program more uniform and more 
usable by various administrative levels. The Commissioner of Education is 
requiiped by Section 1^51 to develop standard* evaluation models, regulate 
their use, and provide technical as^istahce to support both tMat use arrd 
the iniproveiTient of. Title L evaluation in'general, During th/e nearly four 
years i^ince the law was passed, USOE has developed three ajrernattive modeTrr' 
' consul ^d extensively with^ State and local administrators; revised recomm^da- 
tion$ as appiropriatej conducted na,tional traioing sessions; established ten 
centers; to provide free^ on-call assistance' on Title I evaluation matters; 
, and distributed reference materials and pamphlets on specific topics (jf 
r&levance (such as administering tests, interpreting scores, etc,),\ 

Although the general reaction has .been largely positive as evidenced by 
the voliirttary use of thcf proposed^ metiiod^ by nearly 40% of the nation 
Title I districts, the reality of impendTng program regulations requiriipg usage 
of the models h&s caused some objections to surface, Specifically, the Council 
of Chief State School Officers has. commended USOE on the extent of cooperation 
with States for the past four years but reacts negatively to the idea of a 
national mandate, Although their reaction often contrasts markedly from that 
of other staff in thejr own/SCates, we feel a'need to continue soliciting input 
^ffom all groups, ■ 4 




■ ' ' L. INTRODUCTION 

I 

* — ^ y - • .J^ ^ ' 

The passage of Title I of th6 Elementary and Secondary Education Aft ; . 
of 1965- represented a new direction. in Federal education legislation fn 
several^r^spActs-^— I4;s^ba^ to rec±tfyjthrough better schooling the <i1s- 
advantagement of children from families living in poverty-, its authoiMr f 
zation for millions of dollars to school districts serving Targe numbers" 
of such children, and its requirement for repeated examination of^ the 
impacts of the serv^lces funded by the program contributed to its uniqueness. 
In the thirteen y£ars sinca Its passage^ these factors have continued tb 
have an enormous impact on education in the nation as a whole. 

In the'last foyr years'* especially, the evaluation component of 
Title I State programs has received much attention as an outgrowth of 

. Congressional ,^FederaT, State^ and often public frustration with the 
qgality of the data available to describe the program's Impaets. Many 
changes have been implemented at all levels as a resujt of /this frustration, 
and they* seems to be reason for optimism at this tiilie regarding data quality 

' in tbe future^ 

The purpose of this report is to^describe the status Of the changes 
lUnderway in districts. States, and Washington, D.C. as a result of the 
addition of Section 451 to Title I in 1974* Underway since the autumn of 
that year, the program to facilitate these changes consists of work to 
standardize local and State, evaluati^^n practices^ ($lso recfuired as a^ 
secondary effort are nationaT evaluat-i/)ns of program impacts, but they 
wtiU^nbt be*iscussed ip this report. Section 1 51 requires^, that the 
standardization of evamation practices occur through State, ^nd local use 
of models developed by JsSOE aAd theft technical assistance be available to ^ 
' than to facilitate the use of the models'. 

The 'program is administered by the Office of Planning, Budgeting* and 
.Evaluation in the U.S. Office of Education in its Erementary and Secondary 
Programs Division. Aljo'involved are personnel from the Division of Educa- 
tion for the Disadvantaged, the .office whiqh administers the Title I program 
federally. -TJie Nati(Mi?il Advisory Council on th^ Education of Disadvantaged 
Children arid personnel in the regional offices also advise OPBE regarding 
this effort. Figure' 1.1 illustrates the relatiooshlps ampng the evaluation 
ahd program units, the ^'Section 151 group," the States, contractors, -and other 
interested' organizations. 
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The modelSt in brief : ■ . 

, The Coimilss'loner. Of Education is required by Section 151 to develop 
mocifeTs which "specify objective criteria," ''outline techniques,, .and * 
methodology,'* and produce; data which are "comparable on a , statewide and 
nationwide basis*" The tenti "models,** then, refers to sets of procedures'* 
regarding data collection, analysis, and. interpretation which when "followed 
will yield^estiinates of progr^aip, iiftpact on participating youngsters* The 
steps leacling to such estimates ar^e often complicated* 
* ■ ■ _ 

The perplexing question in all evaluations of social programs- is to 
what degree a noted ilriproyement in conditions (an increase in childrenr's 
reading scores, perhaps, or a decrease in traffic fatalities, increase inj 
voter registration, drop in number of influenza cases, etc*,) i^ really due 
to the new strate*gy tried* Other influences on those conditions prohibit ^ 
an administrator from knowing how efficacioi/s the new strategy y/as, and 
steps must be taken to sort out from the overall change the effects of 
tho^e "external" infjuences,-. The "remainder/' then, is assumed to be due 
td the administrator's program/ 

The *'sprting_out" mentioned above is the goal of the methodology 
, embodied to the evaluation models, "Influences, on childrenr's educational 
perfontia^e other than the special remerfial help tjifiy receive through. 
Title ^ include their maturatiorv, participation iji a regular school program, 
greater familiarity" with the tests, etc., so 'th? steps in.the evaluation 
modjgls tnust allow the administrator to determine the degree to which Title I , 
h^ helped the children over and above the improvement they show due" to the^^e 
lather factors* The rationale behind all the models is on^^of observing over 
a period of timethe progress of other children who have, presumably, been 
exposedjto the yarious "other influences" on their educational performance, 
but have not been exposed to Title I, The difference between their progress 
and that of the program participants can then be reasonably attributed to 
the Title IvS6rvi<;es, ^ . ■ ' 4 

T]ie three models, therefore, prescribe pre- and post-testing for the 
Title I children as wetl as for a secopd group. For convenience, the children 
in the tgst publisher's nontiing sample can act as that second group in the 
simplest model* (Model A)v a local group of children similar to those in ^ 
Title. I but who are hot in special compensatory classes comprise the comparispr 
group in Mod^l Bi the children ift the same'grade as the Title I group who*y^ ' 
scored higher on ^ pre-test than the Title I selection score provid^ the bfl(sis 
for comparison- in Model G,^ If various conditions A^e met, statistical pro- 
cedures allow the use of the data from these catiparfson groups fjcfr "soj^ting 
out'the effects of inftuenceSv which\rival participation in Title I^explafn 
Title I students 'improvement* 





* In short, to use one of the evaluation models, ojie (fiust test the 
Title I>-stucjehts twice using an instrument selected because its content 
corresponds to the project objectives and assess the amouht of student ^ . 
growth. Depending on the nature df the similarity between the students 
in *the Comparison group and^those in Title well as other factors), ' 

one uses an appropria'te statistical procedure to yield an estimate of the 
Title \ students* progress, had they not received the ejttra help. This 
estimate is subtracted from the figure reflecting their actual progress,' 
and the difference is attributed to the Title"! project. ^' ' 

^ Although certain procedures must Be followed to, make the estimates 
valid, t;here is much flejtib^ity in the Title I evaluation models to allow 
locabuse of preferred tests, liifferent scales, and most appropriate 
scheduling of the testing* For\eporting to States and USOE, the estimates 
of project effects must bfe expressed a metric common across sites. That' 
*is done by conv^ing the local figu^es^o a scale which incorporates a 
national distri.bution of scores (and far the different models, that conversion 
can occgr at various times in 'the "process). The'n^tional scale*has been called 
"Nari^l Qurve EqujvalentSt" or more, frequently, **NCE*s." 

Overview of this report ^ . _ ■ ^ . ^ . 

The, remaining chapfen^.ia this Hfceport describe the status of the use of 
th6 models, the experiences bf people who used them during the school y^^f* 
1976-77 , USO£ progress in "drafting regulatidns regarding their use, and* the 
.na-ture of work planned for the near future to improve the quality of the data* 
, produced. ^ , - , 

Specifically, .Chapter 2 Include^ a brief histoid of the program to^ 
develop- models and encourage their use* The work 6^ the Technical Assistance 
Centers, the numbers of 'districts using the mode\s last year , 'and projections 
for next vear are also discussed. - * , , * 

"^-^ More detail about the use of the models last year Is the focus of ■ 
Chapter 3. Activities at both the State/and local levels are descrljjed as 
well a^successess and problems reportecr by people in those sites. A list of 
recomnffidations for further refine^ient joI materials, etc., is included, ^ 



^ Simultaneous with the adoption of the new technique^ by many States and . 

districts is the process in ySOE to make thaif use marraatory by^all Title I 
girantees who Qffer services 1n the basic skills to school children In grades > 
2-T2. That process,* the public input incorporated Into it, and the Initial 
ideas about cgntent of regulations are described in Chapter 4> 

\ . \ * . . 

Chapter 5 is a summary^^of conclusions about various aspects of the 
Title t evaluation and reporting system and the program to support its use. 
Plans for future work in this area are also described briefly. 
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CHAPTER liy, STATUS OF THE IMPLEMENTATION OF SECTION 151 
rfn^ 



. Ill the^pring of 1J974, members of the Committee on Education and 
. Labor of the U*S. House of Representftives^ decided that the Office of 
Education should assume a more visible leadership role 'in>the evaluation 
of the effectiveness of Title I of the Elementary and Secondavry Education 
- Act of 1965* They amended the Act, adding Section 151, which defines 
various^aspectfi of that increased lesdjership role* > , ^ 

Wof^k of vanous types has been underway for the four years 
' since the formulation of Section 151, and the pgrpose of this chaptenr 
Js to describe the current status of that evaluation program* Ttiree 
sections will follow: the first will surmiari^e briefly the history lof 
tbe Section 151 program; the second will" descrilje some^of its effects 
in terms of the numbers of LEA|s nationwide using the new Title I 
evaluation techniques and the range of Activities underwity by Technical 
Assistance Center personneV; the third will discuss our. expectations 
regarding use of tbe models next yestr. . " * > 

. HTstory and early years. of the Section 151 program 

Figure 2.1 on the foUowiog page illustrates the time-line for tbe 
\,work in. tISOE to standardize the evaluation and reporting activities of 
ESEA Title I grantees* Early efforts to try to use SEA evaluation data 
^were frustrated by their lack of . uni.formity, and suggestions were made 

* as early as t972 tdat 'U§OE should develop, methods for their use* 

* m 

Reauthorization hearings'in the^spring of 1974 (and the subsequent 
, passage of P*L* 93-^80 adding Section ISl to Title I) strengthened the 
ihterest in those suggestions, and work was begun to develop models, ^ 
Prototypes were reviewed ip the spring of 1975 by USOE sta'ff, members of 
, tw6 , evaluation panels, ^and SEA'staff ^in the stales* Th6'ir reactions, in 
general* were that the models looked promising, and USOE initiated a new 
project invd?ving visits to all States (and at *least three districts of 
each)* ^ ; ' . ^ 

The^vjsits (each Sibout two weeljs long) continued though June of 
1976, and national workshojis were conducted that fall to Vain State 
personnef in the new procedures. Also in attenjiance^at the workshops , 
.were staff of the Technical Assistance Centers. established under contr^act 
' to USOE, in September of^l976; ' . ' . \ 

The purpose of , the Centers is'to provide services to Stajes; and, at. 
" their discretion to local districts,. to"help tbem addres^s uniqu6, site- 
specific Title I evaluation problems. Their Work extends the more generarl 
advice provided to Stat^ personnel at national training session^* Hfenge, 
following the workshops, the initial task of each CentecWas to reach a 
written agreement with each in its, region concern iflfl ftbe kinds of ' 

* services to be pravl<Jed $nd the best ways of 'providtngRese services*. 
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FIGURE 2.1 Chronology of evaluatl^ model c|,eve1opment and use 
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1971. Need Identified for a major 
evaluation of Title U but there 
v^tild be a long wait for results. 
Data from State reports would be 
lyzed In the Interim, ContraclK 
to do '%o% 



, /Data fr 
( analyze 



1972. Results from attempt to 
analyze. data were that they were 
nop comparaWe across States arfd 
often not'valid. 



1974, Re-authorlzatton hearings for 
T|tle I; Section ,151 was added; ' 
contract signed for'the analysis 
of State reports and development 
of eva'luatlon models, >^ 



1975* Required report to Congress 
about ^Section 151 subi^itted as due 
on 1/31/75;* results from-effort to 
develop Inodels reviewed by USOE» 2: 
panels* ^nd ^Jer^onnel-^ln 9-States^ 
Based on favorable rea,ctions» 
dec'islon was to vislt'all States 
and at least ^3 LEA's in each. Visits 
bedan 1h July. 1975, . 



1976- Visits to all SEA 's and over 
4jQ0^LEA*s were completed in. June. 
National workshops, for SEA personnel' 
were held in the fall* Technical 
As^iKtance . Centec staffs began 
work' with SEA*s, LEA*s in about 20 
States began using the models. 



1977, About 40% of the nation's 
Title I districts usied^ the mcnie^s. 
USOE discussed the exper^iences 
with those who had used the mode\s 
the previous year. ^ ^ 



1976^' Twenty-three States plan to 
have all. their LEA's using the 
models for school -year 78-79; 
training by SEA and tAC personnel 
wfll continue A iHemouandum to 
all States and dr^ft rules will 
be distributed.. 
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In ordfer to reach this agreement* SEA personnel had to decide which 
office within, the SEA would be responsible for negotiating with the TAC> 
and which services would be emphasized. This often tnvolved-an examination 
of (1) "areas of respCmsibility within the SEA; f2) SEA-LEA relations; (3) 
areas of technical expprtf^e of SEA stiff members and local ichooldistH 
talents; (4) the role of Title I evaluation ih the overall goals of thfe^SEA; 
(5) availability of funds- for Title I pnd related programs; and (6) tne de-. 
gree to which changes, if any, woujd hWfito be made to meet the ip^quirements 
of the new system, A liaison person wasTSjlentified in each State as the 
individual tfirough whom the TAC would clQftf informatfon and activities, \ 
Next, the. TAC's ^ind State -personnel negotiated a Letter of Agreement re- . 
^ar^ing the activities to be, undertaken and the procedures the TAC was to 
use in responding to requests for assistance, 

Dup to the complexity of the decisions which SEA persdnnel had, to make / 
before a TAC could begin to provide services, reaching an agreement often / 
took sever^armbnt^is, even in States where the personnel, were extremely re- ^ 
ceptiv^-4o-4be^^^|f^^ techiii'Gal assistance for £SB\ Title I evaluation, 
(For those SEAfi which whpse 'initial reaction to federal technical assistance 
was .somewhat le^ tf)^, enthusiastic* reaching an agreement took longer* and 
the provision5i£^,*5eryices often began slowly.) Thus> the TAC s^ had little, 
i*f ^a|jy» impact on'the,<tesigns of th^ Title I evaluations used during school - 
year 1976-77> Iji^t, ifi some instances* they difjLiJfovide help in interpreting 
results in disti^icts^which used, one of the evaluation models that year, 

er of LEA^s using the new evaluation iprocedures - 

■ t ' \ 

; Tffble 2,1 is a list of the percentage of districts, by State, using one 
of thfe reading and mathemftics evaluation models in 1976-77. (The reader is 
warned: that the use varies across sites with respect to quality ^and thoroughness 
As later discovered, the models were often attempted, 'but not. completely 
followed, ^ Nevertheless* the informationi provided in Table 2.1 can, be used as 
a "tiaseline regarding attempts to implement' the Title I evaluation and reporting 
system .during the 76-77^school year as well as currently. (For cftmparison 
purposes* also listed* in the table ts the percentage of districts in each 
state attempting to implement a reading ana/or mathematics Title I evaluation 
mojJeKin school year J977-78, ) r-J ' . 

By far-, the vast majority of ,di5trtclfs currently" implementing the system 
are us^ing the norm-referenced model* Fpr/the jnajority of Title I projects, 
this is the .only model that will (irovide jdata ^hat will meet the requirements, , 
tff the system and provide meaningful resJlts at thfe local level* Host projects 
(jo not have enough stutjents to pui;sue a control (comparison) group model or a 
regression fllodel. In- addition^, some disA^ricts have chosen to use-a^horm- 
referehcerf model because they7eel that /it is the easiest t6 implement 
(although this is not always true)* 
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' Table 2,1 
Approximate % of LEA's Beglnnttig to 
'Implement an Evaluatton Model 



RMton I ' ' 1976-77 

Conn - 140 Districts 
Maine* 164 
Mass.- 340 
N.H. - 165 
R.I. - 40 
Vermont ^-54 



Region 2 



X^s 



N.J. - ^Sa OlstHcts 
N.Y. - 7'SO 
Puerto Rtco - 30 
Virgin Islands 

Reg-ion "S' 

Oetaware - 23 Hist. 

Maryland - 24 

Penn, - 505 

W. V«. - 55 

D-C. - 1 

Virginia - 140 

Region 4 

A^a.Vl27 Districts 

6a!* - 187 --T-^ 
Ky. - 181 
Hiss..- 152 ■ 
N. Car., - 145 
S. Car. - 92 
Tenrf. - 147 
Region 5 

* 

111. - 797 District? 
'Ind. - 302 
Mich. -575' 
Ottio - 625 
Wise- 388 
Minn.- 439 



100 
20 
1 
0 
2 
4 
100 
2 

0 
0 
0 

100 

0 

'o 



1977-78 



25 


•54 


97, 


.100 


94 


loo 


100 ~ 


100 


1& 


50 


93 ' 


100 




• 9 




5 


100- 


100 




lOfl 


■78 


100 


•0 


^' 12 


0 


. 2 


9 


100 


100 


, 100 


0 


■ 29 







100 
35 
15 
2 
30 
20 
100 
IJOO 



7 
40 

0. 
lOCT 
. 15 
13 



teg ion f 



1^76*77 ' 1977-78 



Jlrk^ '• Districts 
■ LA.. - ,66 Parisijes 
M«ix<-86 Districts 

Okla; -613 

T6xas-I0l7 



Region 7 



100 
,0 

or 



100 
10 
.22 
100 

1 



Iowa * 440 OtstHcts 


9 


45 




A 

? 


J UU 


*«ssouH- 389 ' 




- 70- 




n . 

u 




Reqton 8 




* 


Colo* - 172 Otstrtcts 


0 - 


7 


Montana -250 


50 


, * 100 


^So* Dak*- 267 


0 


25 


So* nak>- 188 ' 


100 


100 


Utah - 40 


0 


20 


Wyoming - 42 


0 


30' 


Reqton 9 






Arizona - 145 Otst* 


5. 


36 


CaK r 1080 


0 . 


• 1 


-Guam - 1 


100 


100 


Hawatt - 7 ^ 


0 ' 


100 


Nevada - 17 


0 


- 100 


Samoa - \ 


0 


100 


TrusrTerr*-6} 

4 


0 




Region TO 






Alaska - 48 Olstrlcts 


0 


' 19 


Idaho - 105 * 


98 


• 98 


Oreg6n -265 


* 0 


- 9 


Masb. - 290 


.0 ' 


35 



■ Toca Is Total LEAs - 13^576. 
lSt76-77 TocaL L£Ae Beginning ImplemenCaclont- 3L21 

Percenc LEAs Beginning Implemencaclon; 2111^ ^. 

'fl977-76 TaCal LEAs fte^lnnlng IttpLemenCdClonf 5386 \ 
, -Percenc ,L£A3 Beginning IntpLemenC^clon; 401 
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In general , the regression or; coritrol {comparison ) gfoup^ifodel* Is ^ 
moU 11ke\y to be used by a meaium to lar^e pfoj^ct, wfth large-scale ^ 
.testing program in place, and a computer or sophisticated Wrroscalculator ' 
availat>Te. Id some ca^s, where a regression or control* (compa^son) model 
fs appliednthe distripi^ns also to -use a norm-rfeferei^ced model. {Districts 
have been encouraged « design their evaluations to conform to the requirements 
of more than one model, in case they are unable, suddenly to- meet the require- 
ments of one of 'the moiclels^. v^- , • , . - ^- 

The TAC's have^been afotively involved in providing workshops and on-site 
coD5uUatio>)- to States^af& they implement the^new evaluation procedures. ^As 
stjown below, the earVy^workshops focused o^n orienting LEA and SEA Tit1|| I ' 
staVfs to the models. Following the orientation workshops* the ^'$ual pattern* 
was to hdld more focused.^wOcksTilp^s on particular topics for smaller groups, 
e.g., test selection,, implementatipo of a model and rfata, analysis procedures- 
Table 2.2 lists th^i" perc^enta^'e of LfA'S in.a state that have been trained; 
that is, those who have^e$ft;)resented the USOE system by TAC staff members or 
SEA staff members.' The 1 ist - belo>t Uken from the Region VIII 15-month report* 
describes the workshops held ^d the nature of SEA and. LEA consultations. 



Workshops : - , , - . ^ 1 



A* Ortenta^feibn— -overview of reporting system.and the three evaluation models 

' ♦ ^ 

&* Model' Selection--procedures .and considerations when selecting a model 

C. Model A rmplementation--details of how to use jt , « ^ 

D. Model e-Implementation— details of how to use i£ ^ 

E. Test Charac*erfetics and SeUction—xharacteristics of test' to considei^ 

. '. when selecfing one for project 

. . "'^ .evaluations ' 

F. ,Data Analysis— details orhow to doHt - ^ ; 

" ' • ■ ■ r 

Con sul tatioo ■ - - ^ , 

'■ ' \ ' 

A. Pilot y te Decisitirvs— ^eprding the appropriateness of LEAs serving 
,4 " . • ,as pilot sijes- for a specified model (information 

• ' ^ 'generated from their experiences would be used to 
' ^ ?*ssist/Other>«listricts) 



Table « zJ 



■lOi 



•% H LEAs Rece 



Some Training ats of ]0/]b/n' 



■■ REGION I . niSfRICTS 

. ■ . (as of 12/31) 

.Connecticut" ^ 140 ' 

"Maine « ^ 164 

Massachusetts , 340 • 

New Hampshire 'IfiS 

Rhode Islands ' 40 
'Vermont • , ,. ,54 

^ RDSIBN II; • '* 

New Jers^ . ' '450 * 

New York . 75.0 

Puerto, Rf CO • 30* 

Virq'in Islandc i 



REGION III (as of 12/31) " 

.Delay/are *■ 23 .100. 

Maryland 24 . - 20 

Pennsylvania 505 ' SO 

Virginia . ' 140 35 

^ West Virginia 55 • -100 

Dist. of Col:'; • .1 TOO 

' REt^rON IV (as of 12/31) ' 

Alatalf^ 
FloHda 
^Georgia 
Kentucky 
MissiS^pi 
N. Carolina - 
S. '.Carolina 
Tennessee 








127. 


' 100 


67 . 


100 


. 187 


100 


181 ■ 


n 


152 . 


ion 


145 


100^ 


^ 92. 


ion. 


147 


IQO 




Tot^l/i lU.relvlng. 



REGION V 

Illinois 
Indiana 
Michigan'- 
.yijinesota 
hio 

Iscpnsin 



797 
302 
575 
439 
625 
'388 



DISTRICTS 

100 
• 0 
59 
64 
51 



(ifGION VI ' 



ArVansav 

Louisiana- ^Parishes 
^fewylexico - 
Oklationia 
texa 



REGION, VII 

Iowa 
Kansas- 
iiissouri 
Nebraska 



REGION VIII 

Colorado 
*^ontana 

Oakota- ' 
S. riakota % 
Utah 
Wyoninq, 

REGION IX 

♦ 

Ari zona 

California • ^ * 

Guan 

Hawaii 

%vada 

Samoa 

Trust Jerritories 

REGION X 

Alaska 
Idaho 
Orfeqon 
Washln^lton 



585 
'466 
86 ■ 

613/, 
• i,017 



440 
250 
389 
300 



100 
100 
22 
100* 
5 



77 
100 
55 
75 



172 


7S» 


250 


90 


267 


90 


188 


95 


40 i / 


75' 


42 A 


. '75 

* 


145 


100 


080 


0 


V 


100 


7 


IQO 


17 


100 


1 


100 


6 


100 


48 


78 


105 


<38 


265 


8 


290 


■ 86 



"ERIC 



17 
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B* Testing DecisionS'-regarding characteristics of districtsSnd of various 

tests , which should be considered when selecting-a test * 

C. Data Analysis Procedures— regalf'iJtng the most appropriate arrangement rar 

data analysis {e.g? , eoniputer versus hand- 

^1 analysis ' ■ ' ^ 

D. /Neglected or Delinquent Programs-r-nieeting with SEA and LEAs to discus? 
I " .evaluation pf'ocedures in the interijn* 
/■ . before models are disseminated by USOE 

E. I Planning 'Future york--ineetin5 with SEA to detemine specific content of " 

'workshops and more general strategies, for training 

. ■ and assisting LEA personnel 



F. ■ Reporting Forms— regarding reporting formats to meet specific needs 

G. Computer Software Instailation--assist"ing SEA-and LEAs to install comput^ 

program for data -analysis 



Table ^.3 lists the special inter&sts of each State and reflects U 
wriety of/The activities of Center staff. Also, -copies of Jhe 15-iiprvtJ 
report (0ct.y 1976-Decem^er, 1^7^) from each of the Centers are available, by 
request/from the Office of Planning, Budgeting , aC!d Evaluation. 



, Expectations for use of the evaluation mpdels next year 

^As shown in Figure 2.1 and Table 2.1; we expect more" widespread us& of 
the new evalui^tion procedures next school .year^ There is much anphasis iii 
, sea's this spring on revising materfafs and training, local ffeVsonnel. 'As of 
February 1 this year, all SEA's but one have a 'plan specifying how they^yiU 
move into statewide use of the system^ and in th^near future, most^^wfll. have* 
achieved that goal. - T ^ ' 

, Other new developments which facilitate adoption of the models are ' 
...regional coordinating councils who meet to dVbcuss commo.n problems'and 
solution/ and the distribution of USOE' s- quarterly evaluation newsletter to 
'a.11 LEAs'. Both of those activities increase the visibility of the Title I 
evaluation program and encourage SEA pqjrsonnel to proceed with the new models. 



O f 
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' Table 2-3 

Special Interests of Spates 
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RE GION I 
M assach u setts 

Merging of'State and Title I , 

eval uations 
^ Needs assessment 
Computer/management data info 

systems * 

> 

New Hampshire 

Pficess evaluation 
Uae of 3ata for decision mailing 
Affective and early childhood- 
/aluation 



1 T 



gitudinal studies 



Verm )nt 



Loigitudinal effectiveness of 



Conn^ ictjcut 



Ea 



tie I programs 



:'ly childhood evaluation 



Process data 
Rhode Island 



ProBlems of aggregati^on 
Test selection; out of 

level testing 
^ttse of data for . program 

Improvement 

REGION 11 ' . ' 
N few Jersey 

Adapting Model C to account 
foy> local variations In 
achi evemefit grovrth in large 

'LEAs 



\19 



J- 



^ont . 



New York 



State centr'alized processing' 
of data 

Puerto Rico ' . 

Test development, norming^ 
■ equating. 



REGION III 

District of Columbia 



Evaluation design , . 
Development of local norms^ 
Use of criterion-referenced> 
tests 

Simultaneous Implementation of 
all three, models 

Development of a -laboratory 
handbook 

Marj;la_ntf^ 

Test selection ■ 
Presentation and utilization 
of test results" 
Data collection procedures 

,Vi_rq1_ni_a 

^Evaluation- design / 
Test selection , / t 
-Data collection procedures. 
Utilization of test results 
Presentation af results to the 
publ 1c ^ , , 
Overview of evaluati^6n for 
Parent Advisory Councils 
Installation of computer pro- 
grams 

Use of criterlonTreferenc^d 
tests 



1^ 



V 



■\£EGimLi.n (Cpiftj,-*, ".■] 

^ Pennsylvania ^' ' i ' V 

iJlstallation :bf comptiter- programs 
• ' '^Rev,ie!vf or currently aVaiVable 
5;, tests 1^ ; . / ^ ^ 

Test'selectibq.^ 

West Virginia - ^ 1 

* Evaluation design . 
&ata-^col]ection^ procedure^, , 

' Asslstijig ^in the .development ■ 

' olf'.'a.c'oApetency-based reading' 
test for use in Title I 
Assisting H* Virginia in develop- 
v,ing'^ Titlfi I Evaluation ' . 

^ handbook ^ 

De'^l aware 

^ # 

Test selection & reporting forms 
Repprting to parents 
Tnservice training in evaluation 
methods 

Affective measure^ment 

REGION IV 

F lprida 

Computer analysis system 
■ ^ Model C 

G eorgia , 

Affective measurement 

North Carolina ' , ^ 

* 

Affective measurement 
^ South Ca rol in_a 

Techaical issueisi^ in Models 
Tennessee 



Computer prograimVng 

Kentucky ' ^ 

Genera! orientation .to systems 
^ Jlisslsstppi ^ 

Staff training; 20 
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RKJjl)FxV (cont.) 

Al abaina 

Early childhood evaluation 
Functvional level testing 



REGION 



Illinois 

r ^ ■ ' / 

.COBOL cWiputer grogram 
More detailed test inforniationj 

relate evaluation to state- 

assessments . > 

Indi ar^ 

Include grades 1 and 2 In 
evaluation- 

Mi^ch i^g^a n 

Reduce norin referenced .djata . 
collection burden for^districts 
using ORTS. Coordinate, 
evaluation. with state assessment'. 

Mi^n n^s qt a ^ . ^ 

Install computer program" on 
Burroughs* - 

Ohio . ' 

Include grades 1 and 2 i'n ^ ^ 
evaluation. 

♦ 

Wi scons l^n 

More detailed test information. 
Process evaluation 



}cess eva^u 
[0 NJA \ 



BEGION 
Arkansas 

Test administration 
Test score Interpretation 
Model Al implementation 
Migrant evajuatibh 



RE6IDN VI (Conti.) 

Louisiana ■ , 

Non-nomed testing 

Awareness of PACs 

Computer support - 

Hand calculators . - , 

Objective writing '^"^ 

New Mexico * ; 

Reporting fonijs / 
Model Al implementation 
Affective measures j ' 
Regression models 

Oklahoma 

Computer support 
Data analysis 

^ Texas 

All models Implementation 
Migrant evaluation 
Essen]t1al of good process 
evaluation; Neglected 
and/or delinquent models 

REGION VII 

Iowa , 

Installation of computer system; 
Special study on quality control 

. Nebraska : s ' " 

Installation of computer system; 
Special study on base-line data 

Kansas 

— i 

. Computer program documentation 
Missouri ' ^ 

General* orljentatlon to system 



REGION V-Iir - 

No. Dakota ' 

X/ ; 

Early chfldho^^ ^^aVaatlon 
SOt Dakota - 
N/D e'it^luji^n 

' T'. ■ ■ 

Using "affe^itive measures 
Use of jit'^e I evaluation for 

local detl&ion making 
Integrating Title I Evaluations . 

witn-State data collection 
Test' selection issifes 
Process evaluation 

Mon tana ^ ' y 

Use of CRTs and local horns 

Colorado 

N/0 evaluation and CRTs 
Jligrant evaluation 
Process evaluation 
Coordinate Title I and other 
testing ... 

' Utah ^ ■ 

Computer System 
Evaluation involving minority 
.groups; esp. Native American 
CRTs and local norms 

J^EGIQN IX 

Arizona • 

Local nontts 

Regres^sion problems (Model C) 
Combining NRTS with -non NRTS 
Upgrading staff skills 

CaHjfpniia ' ' 

Model C and A pilots 
Combining Calif, assessment with 
•Title r ' 

■ 21 ■ 



REGION IX (cont,) ' ■ 

Hawaii * ; 

J. Local use of Title I resi^ts^ 

J'r^ocess eyal^at>on_ _ 

Test seTectlon ' 7'' 

Nevada ^ , - 

Modification of computer output to 
get individual scores 

Guam * . 

InstrumeryNrtilSrrproblem^on TESOL^ 
Selection and development of \ 

instrunlents on TESOL ' : ^ 

Improving selection & placement of c. c 

children / ' 

Trust Territory 

- Developing own tests and local norms / 

Samoa ^ - 

Develop plans to coordinate Title I 
and other Federal programs eval's - 
Improve evaluation capabilities : 

REGION X 

AT aska_ ^i * ^ 

m 

Developing local norms ' , 

Idaho • 

Improving quality of Title I eval # ; 

Oregon ^ ^ ^ " 

Establishing computer network 

Use of Rasch Scaling in Title I evaluation, 

W ashington 

Iiitprove staff evaluation capabil iti^Sv 
Expanding utility of Tittle I computer prtograms 
for feedback to teachers 
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CHAPTER III. EXPERIENCES THE USE OF THE •NEW'EVALUATION- SYSTEM / ' 

Lie purpose of this chapter is to .describe (1) the experiences Qf s , 
Stateflmd local personnel 'who used the evaluation jjiod^U to assess project 
effects for the 76-77 school year, (2) the successes a^nd difficulties they 
exberienced, and (3) various Suggestions (from them as well as fr:ora others)^ ^ 
'fdr m&king .the procedures less difficult, To put their experiences in 
context* we will first outline the'steps necessary at the State level to 
^et thd'models used by LEA^S- Secondly, the data manipulation prescribed 
^by the models, themselves, will be described^ sufch that users' suggestions ' . 
/for refinement, can be related to the p vera 11 process-, ^ 

Hence, th^is chapter^as four sections: (1) the SE^T^rocess ^to facilitate 
LEA use of the models; (2) the data manipulation necessary in project evalua- 
tions; (3) the experiences of personnel at the SEA and LEA leVels in adminis— 
tering or.performirig the evaluations; and (A) suggestions for modificertions . 
in the USOE - recommended procedures- 

The SEA protess to facilitate LEA use of proposed USOE evaluation models 

f 

During the 1976*77 scliool year, 28 of the SEA's across the nation had 
LEA'S using one or more of the proposed models to evaluate the reading and/or 
math outcomes attributable to participation in a Titled project. (See Table 
2:1) for the most part, those sites used the techniques for Ihe fijfst tjjng- 
Information gleaned from the evaluation personnel there gives an i^lea/of the 
administrative procedures involved in effecting the change to new methods- 

0P8E personnel contacted six of the SEA Title h directors who have had 
the longest and most compressive experience in using the new procedures* 
Without exception *they were willing to share information concerning the 
characteristics of the' implementation process* Visits were made to the . 
states of'Rorida, Iowa,^M3iiie, Ohio, South Carolina, and South Dakota*! 

The generalize step-by-step procedure-followed by a SEA is presented . 
^ below, and a GANTT chart (Figure 3*1) illustrates typical timing of those steps 
The procedure does not represent a specific state, but rather a composite of 
what 'happened in'the States whqse staffs we contacted- ' ' 

1. SEA and some LEA personnel Jxi every state learned of upcomintj changes 
through USOE presentations at national meetings, visits to the. states by RMC- 
Research Corporation personnel (under contract to USOE), and through visits 
by various* USOE staff members. For SEA personnel*, this "first .exposure" tp 
some of the new ideas could have been as ear'ly as summer of 1974 (USOE was, 
alreacfy recommend iiig the avoidance^ of certain test score metrics in conjunc- 
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.> * F.igyre 3.l' - ■ * ■ - 

" Sequence and timing of the implementation steps in a typical state. 

* V * * * ^ ' 

' • Time period: '< ' 

■ ' School-year 1974.-75. 1975-76 1976-77 • , 1977-78 

* Steps:' Mpnth 6 9.12 3 6 * 12 3 '6 9.12 3 ^ 9 M2 3 

. 1. SEA learne^l of upcQming changes , , ■ -* 



2. decided to implemeYit them 

3. revised their forms & i'nstructions 

4. conducted orientaftion meetings 

-.5. identified, pilot s^tes ^ ■ 

. 6. held additional meetings . v . 

* ■ . 

7 LEAs modified student selection methods 

8. decided level^of dataTcollection \ ■ 

9. SEA assessed their progress 
10. expanded the implementation 
Tl. collected ev.jp'UatlPsn, reports. 
12. edited the data' * \ 

\%. ag^regat®* data for USOE 

14; further analyzed data 

'l5i identified exemplary projects/ ; 

16. revised the. program . 

17. medifled their evalifat'ion techniques 
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tlon with the effort to document, and disseminate information about effective 
practices), or as recent1y-^"s the 1975-76 school year. 

^ 2* In* some cases, SEA jtaff decided to pilot, field test, ot ftfUy 
implement the new system* ocGasionjilly with linfitations on their LEA's as 
to choice of model or test. Evaluation'^nd program personnel discussed the . 
ramifications of adopting the proposed models, balancing technical and politicaK^ 
considerations ag&inst the expected return of useful information* In some Statei* 
these discussions disclosed a preference for Statewide use of dr single nwdel * 
In others, a decision was made to encourage the LEA's to consider all tn|^ models* 
and choose the ofie(s) best suited tadistrict characteristics* 

3* Following the SEA decision to adiopt the model (s), staff members revised 
their evaluation reporting forms used for collecting data, from LEA's* This re- 
vision w^s based on the types and extent of information required for implementa- 
tion and for quality corttrol* Genera^lly they attempted to reduce the paperwork 
formerly required from the LEA; starting with the draft forms produced urtder 
contract for USOE and sent to SEA's for review, th^ added sections dealing with 
specific "material needed by the SEA (such as data concerning the nati^re of the 
projec^, whether it was pull-out or an fntact class, if aides were used* over- 
all progfam er?phasis> etc J Even with these additions, however, the result was - 
fltn abbfreviat^d ,reJ)orting form compared to those from prior ye^s. 

4* The State then conducted orientation meetings to publicize the change 
of procedures, to train local personnel regarding the new forms, and to obtain 
information on problems which couli^ arise. Material covered at the meetings 
^included an orientation to the evaluation models, discussion of the proposed 
LEA report^ forms, ajid notification regarding the implementation timetable* ' * 
State personnel stressed the reduction of .paperwork (and possibly also of 
testing) as a compensation to the LEA's for the increased rigor denfended in 
evaluation* - . ^ 

• - ^. 'I 

, At all the workshops* attempts. were 1ti$de to keep the number of participants 
to a minimuJn (to allow interaction 'among them and to allow attention to each 
periWn), This meant that since attendance of at least one person from each 
LEA was, compulsory, States'had to hoTd several workshops over a per*iod of weeks* 
dften the session^ ended with the SEA distributing a questionnaire regarding th? 
attendees' reactions to the changes,, thoughts about, other types of workshops or 
:assistance they, might require, and provable choice of model. 

5/ UsuaVly SEA's identified pilot sites to try the jnodels the first year 
(75-76), Since tf^e workshops genecally took place after the selection of 
Title I participants and tlieir pretesting in the schools that year, the fySl- 
flejlged system could not be started immediately. Consequently the SEA chose a 
subset of the schools in the State to pilot, the models, .and to use the new forms 
and instructions ff they desired. The data collected fran the pilot school * 
districts the first year were generally somewhatj 1 awed for; proje.ct assessment 
since Most of thfe L^'s had problems following a]l of. the procedur.es for any 
given model . Th^'fexperienc* was considered valuable* however,' as a basis "for 
determining remaining difficulties, most troublesome requirements, and furtfler 
needs for technical assistance. 
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■ ^ 6. Coincident with the;prbcess of cfi^osing try-out" schools > the SEA ^ 
held aclditlonal meetings . Technical matfeHaTs were distributed to all pilot 
L£A's> and their participation requested at area workshops, SEA personnel., 
by this time much more conver^nt with t^^B new procedures , gave detailed 
instructions on the model requireojents* They were often assisted by Technical 
Assistance Center personnel {as descried in Chapter II), 'THe St^ite leaders 
helped LEA evaluators try out actuajjj.janalyses on sample data* Those .Slates \ 
in wbich the SEA prt)gram perspppel^layed a majon, visible role >n tJie^^Ejrk' 
shops fpund that their early involvement" increased the chances of suebess. 

7, With Sffl encourageffient both ^n the meeting's referred to above arld^ 
in frequents mailings of technical papers to the schpols, the LEA evaluators 
modified their methods for selecting children for Title I services, ^ince 
most of the LEAS chQ:M t$ use^-MpdM A .(the jidrm - referenced approach), they 
wefe required to sfc]^OitT^*1f%aj^ on. something ojther th^n their 
_pretest scores, (Pc^ffOT^y/a^jtoSt every LEA had selected students, on the 
pretest, thus leading t&2fetifns?^s qf th^ efficacyiOf Title I artificially, 
inflated due to. the r^gress'ior) " efifect, ) Suggested new fl)et;hods incliided the 
use 'of diagriostic tests , failing grades in the classroomV obje^tiv^/ teacher*- 
rating scales, district-develbped tests, dnd subscdres of standardised tests, 

8, The LEAs also decided on the most appropriate level of (fajja^tollection 
for their situatfon. The evaluation system requires that {lata be'collected on 
a '^project;" h^e, any information obtained should describe- th^t^unit. If a 
project were defined on a "micro" tevel $ucf^ ae s ^ single* ]anguage arts class- 
room, the number of students comprising the observations about that unit mighlE 
be too few for stable conclusions; if def itwd on a *'macro" level (for example, 
including all language arts. instruction in a illstnct), perhaps too little de- 
tail would be available for use in identifying the educational practice responsi* 
ble for the change. Thus, a project might include classrooms wntch var1e,d on 
hours per week instruction if the LEA eva*luator conSidered',the tim^ difference 

to be 'inconsequential. Biit, for instance, data from c1asst*ooms in which 
computer-assisted instruction was used would nSt W combined with those in which 
peer tutoring was the mode because the two instructional approaches are quali- 
tatively different. * ^ 

9, At the Gonclusion of the first year of implementation, , the SEA assessed 
its progress thuS far: / , 

a. Had all evaluation personnel in the State been at least exposed to 
the jiew system? ' - 

b* Ha<J a pilot group of lEA's attempted to use the hew methods and 
report forms?, , , ' , 

c, Were^the reporting forms being revised* and becoming simpler to use? 

26 ' : 



Had LEA, procedures for "selecting Jitte"^ students been modified, 
as appropriate, to avoid the contamination of the evaluation 'data? 



! non-pilot-testing LEA's accepting the models (no longer hoping**- 
change wQuld simply go'away)? ^ . ^ 



Were. 

the LiJQiivfc wwuiu ^tmyijf yu anajr ; > 

. ^ f * Was there an awa/eness among the State personnel Qf the need to*^', 
SC'- establish some quality control* procedures'? . ^ - - '\ 

, ' ^ . , * ' ^ ^^"^^'■'^ "v 

10* In the- second scbool year, 1976-77, the State expanded, the implementa- 
^ tipn. All Sttio.ols irf the St^ite were regu^rfed to -jise the models and new repor- 
ting f^rnis* /Additional workshops were pn^ided for those .who wisbed to re/iew * 
the material, an)j^j? new personnel recenffy given responsibility for Title*! 
evaluation* Often, . ^ second workshop during the sch(jol year was ddded for 
discussing tne mechanics oi completing and interpreting evaluation report^* 
* ^ /^^ ' ■ 'i* ^ " " ^ ^ ^ ■ * 

11* At thq conclJHsion of the first complete School-year cycle, the SEA ■ 
collected evaluation reports from* all LEA''s*^ Personnel, in the State office 
scanned the reports* and if they appeared to be complete, they were processed* 
Otherwise they Were sent back to tjie LEA*s for correction* 

In the States with sejiiarate program and ^evallldtion units, at least two 
copies of each LEA report were collected. One copy was given tp the program 
office and thgyother to the evaluation staff* A serious attempt was made by 
the State Title I director to involve the entire SEA staff in evaluation, and 
.to use the results in ^modification of the program^; , ' ^ 

.12*^ As a first task in the analysis, the SEA staff edited the data to 
determine its quatil;/^*'' Th^ general characteristics of the project as reported 
were*" matched with what had been proposed in the project's funding application* , 
Often data from entire grades were missing, or the*numbers of students reporte"^ 
differed greatly from the number anticipated* Appropriateness of tests, forms^ 
and levels* and testing da^ were ascertained* Telephone calls were made to . 
LEA*& With incomplete or erroneous returns L/the^ worst reports were returned to 
the schools to b^^redone* \ • , ' ' ^ 

Then acWil data were checked* This included inspection for out-of^range 
valtjes, excejssive variability Wross projects or grades, a large proportion of ' 
scores in the guessing range, [ncorrect arithmetic, qnusual changes of number 
.of students across 'g^jade^^^etc. Often, the SEA found/that much better editing 
i^as possible when individual student sqores were repoirted to the State* so th^ 
jnodified their forms/ to request! that LEA' s cut studentKjiames off the worksheet, 
and 5end the spores /along wUh xhe report* . - ^ . ' . 

> */ ^ *1 * ' ' 

Apother part df the data qijality control was* the checking of r^esults from 
siJtes-where ejjjrluation was known to be well, done in order to estimate .the range 
of project gaTns likely to be found in the "State* Often such superior sites 
were/larg6 cities where the variability across projects was great* Kence, such 
data were fel t to ^establish reasonable bounds for project tfStimates', if data 
from projects fell outside that r^ge, tb»y were re-examined*fftc, possible errors. 



' 13* The SEA personnel aggregated the most error-free data for their, 
report to, USOE. Copies of the State report were, sent to US(5Frto all LEA^s 
wUhin the stajte, to other SEA's^ and to representatives in the State anja 
federal legislatures^ Sometimes the reports were given to ttje newspapers* 

• : ■• ••• /• 

H* After the required report to USOE was Completed, the SEA further 
- jinalyzed the' data to an^j^r State-specific questions. For example, gjilns 
were ciilculated for projetH:s aggregated by the t;ests used for evaluation in 
orde^ to determine whether certain tests gave an^^evaluation "bonus" ^io the - 
user^- Or gains were calculated for, projects involving pull-out programs and 
compared .with those involving aides. ^ ' ^ . f ^ ^ 

^ ^ ^ ' ' ' / ^ 

1^5. - The evaluators within the State als^o identified exernplary'^btrojects . 
Those with large consistent' gains across grades were s^Ject^d, and if the 
appeared to be reliable and valid, the results were disseminated* Projects 
with large consistent losses were also examined, and t;he State progr*am 
personnel made suggestions about possible methods of Improvemenlj'to the 
lea's- - ' / 

16. Not only! the extr^fces were investigated; based on th6 complete 
evaluation Vesults, State personnel considered revisions to aspects of. the 
program to reflect what had been learned. Where they were certain that greater 
gains were possible in the low^ grades, ''for instance, they encouraged^LEA's 
to emphasize instruction at that level.' If the use of peer tutoring resulted 
in generally larger gains, they stressed that possibility to LEA's forSth^ 
next year's applicatlons. ^ ' iw 

w ■ 

17* )ind last of all> the SEA modified the evaluation larocedures to improve 
the quaflitAof the data and increase, the scope of the decisions thaj^, could be 
guided by tnose^data. Many ofithe specific problems they addressed are delineated 
in the following $ection, find possible remedies are suggested* 

The djjta manipulations necessary in project evaluations 

Jjiere exists tremendous variety, both at state and at local levels, in the' 
ways tnat Title I evaluation information is gathered, analyzed, aggregated and 
,reportedy In the extremes, some States currently gather data at the individual 
studenrifevel from the LEA's aAd. then employ central processing while otheff' 
States rely entirely on their LEA's to analyze their own data: Some State^ 
have automated virtually the entire evaluation process using scoring services 
and automatic data processing; others rely on information that is ^cored and 
analysed' entirely by hanrd. In States th^t have some very large and some very 
small LEA's^ differences in procedures will v^ry tremendously even within the 
State* ^ - / * . ' 



Any explanation for this wide variety q^ractice across the nationa^l 
Title I system would consist of many .fa^ctors — capabilities and suppo,r:t_ 
systems vary widely, different philosophies" and priorities are used to set'"^ 
policies, the numbers of Title I students in a district *()iay range from the 
tens to the tens of thousands, while the amounts of Title I grants might ^ 
range from the tens of thousands to Jthe tens of millions of dollars* .Thus, 
it is difficult to erfvision, at the 'present time, a unique correct way. to do, 
things; the system must te designed to function efficiently with alternative 
methods for data handling, processing, analyzing and aggregating* 



A, schematic diagram of achievement data* flow' through jthe Title I 
evaluation system is presented in Figure 3.2. It is. arranged to display 
the tas^s typically done at the LEA and the SEft level. ^Depending on the 
sop^stication Qf a particular BEA or-^EA, a task (such as conversion of 
scores) can be undertaken at the SEA, LEA or both levels. Wher.e a line in 
'the chart crosses to the 5EA portien of the figure; that represents data 
being- reported by the LEA to the SEA, in the form indicated by the parallelo-. 
gram where the line originates. The roman numerals dejiote phases to be 
discussed below, including test selection and administration, scoring, score 
conversion, data analysis, and data aggregation. . ' ^ : . 

PHase'I - selection/administration of -tests . At this beginning, phase of 
an evaluation, several sl^ps^are iinportant: (TJ the proper test must be 
Vselected; (2) the correct tlvel of it must be administered; (3) the test 
adminlsteration procedures must be'standard; ^nd (4) the testing conditions 
must be appropriate. There aVe many aspects to each of these principles, 
but they can be summarized as follows* 

. The "proper" test is, fbremost, one which.^ measures what is, being taught * 
Much research (Antibruster, Stevens, and Rosenshine, 1977; Bianchini , 1976; * 
Hoepfner, 1976; Stearns, 1977; and Tallmadge, 1977) has highlighted the 
degree to which different standardized tests emphasize different subskills 
within a skill area* Title L evsfluators are advised that the more closely 
their tests correspond to the project' objectives,^ the more relevant the scores 
will be for detecting student growth in the projetet (Fagan and Horst, 1978). 
. This fairly cormion-sense notion is pfteh disregarded as other, factors in- 
fluence test selection. . \ ' <^ ^ ^ 

^ Furthermore, the "proper" test has empirical normative, data^on children 
similar to those irr the project at times of the year corresponding to the 
pre-and post-test dates for the evaluation (if Model A is being used). Also,, 
^the nirrm dat^ are from a representative national (or loc^al) sample of children. 

Using the correct level of tJ)e test means administering, one on which *the 
fewest children ^possible score either at the "chance" level or at ttje top 
score, Ttiis is important bec^iuse a preponderance of the former (magy students 
at the "flooj^"* of Ith^ test) artifically ihflates the gro;jp's pretest average, 
thereby .overstating thetr status "before the project/ An estimate of their 

, gail\ due to thp project would of course then be underestimated • (Similarly, 
if they "top oA" on the post-test, the group's status after the project is ^ 

PpY^restimatej)^; and the resulting gain figure is "again too.smalK) Of course. 
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Ffgure 5.2 Flow' of Data-in the ESEA Title I 
Evaluation and Reportirig System 
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the mismajtch of test level vith student skill levels can also affect evalua- 
tion results in other ways; the important^onsideration is that students' 
performai^ce levels be, reflected as accurately as possible. Use of the wrong 

level of the test precTudes tMs' (Roberts, A. O.H. 1978} Roberts, S., 1.978). 

■* * ^ " ' . ' 

All tfests have standardize^ proceclures outlined for their use. (Even 
"home-made" instruments have instructions for administering them*) Such 
procedures may include timing,- use o,f practiceiitems,, degree, of assistance 
from the test progtors, etc. In order that students' test, scores be com-' 
parable to those of others (especially to the norm data), these procedures ' 
must be followed (Hohst, 1978). . 

lA related matter, is. the insistence on gobd 'testing conditions. These 
include quiet rooms, similar settings for testing project and '^comparison" 
grou6 children, same time of day, etc* (Hprst, ±1978; Tallmadge and Roberts, 
19781. ^ . . * ^ 



Phase II - scoring of . instruments /; This step involves determining the ^ 
test score fQr each ^udent. Occasional ly it is done by hand; - teachers or * 
.^clerical personnel' place a grid over. each answer, sheet, 'mark the incorrect 
' responses,, and^ tally the student's score* MOre o?ten, a service is "used. 
Student answer sW^ets are sent away to a firm who 'v/i IT return lists of 
students tested, the raw score for each, and g^quested score conversions. 
Problems may rise with lost answer sheets, incorredtly coded ones, damaged 
page^, etc. ^ ' - 

J , • ' ' 

Phase III - conv^sion of scores . Due to the mathematics required-by 
the analytical procedures in an evaluatiQn, students^ raw scores (number right) 
must be converted to a standard score retric* In most cases, the preferable 
standard score metric for the confutations is one whjch incorporates character- 
istics of the national distribution of scores for the age group--the NCE. In j 
order to deriye that figure, as many a^s three or four separate conversions may 
be necessary for^^aqh student's score. (A typical sequence would* be ti\e cfijld's 
raw score, to the 'publisher's standard score, to a national rank or percentile, 
to/an NCE). ' ' * ^ . , 

|ome scoring services can pro/i'de all oil these scores; all can provide 
e percerftile equivalfflQts. 'There is a-charge for each additional score re- 
stedi however, so the use of a service foi| al 1 tff the conversions could be * 
expertsive for large proiects* Jhe trade-off is one of accuracy, it appears. ^ 
(See next section regarding comrents fi;om people about their experience?,. ) 

\ V ' - ' ' , • ' 

Phase IV - daty^grtalysis . ' ThiS'is the phase in the Title T evaluation 
where data from individuals are combined into project-level sta^rstics* NCE 
gains, describing the effectiveness qf the Title I project in contributing to 
the student^s' learning above and beyond what is expected "Vrom t}*<."reg{ilar" 
curriculum,, are calculated!. Each model prescribes the^appropriate analytical 
techniques, which r^inge from fairly straight-forward computation^ in the case 
of Model A to complex statistical tnauipulations in Mo^els^ B and " ^ 
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Niase y ' data jiggregation* This is the final st^p ia the Title I 
evaluation, system, prioh to the actual reporting of the ei^aluation results- 
Errors Wade at this stage of the^evaluai:foq are not likely;. to be too 
serious, sinc§ the data on which*the ajggregations are based {project-level 
NCE gains) are usually accessible for, later double-checking. , It may take 
place occasionally in a school disti^/ct central office (vjien a singlfe 
project is implemented in several buildings) byt more often this phase 
occurs at the SEA level. ^ \ ' 

X / . ^ ^ ■, ■ 

The f^rst two sections of jfhis chapter have described the implementa- 
tion of the evaluation models—ViTi terms of'both (1) State administrat^ive 
efforts to bring about a change^ in LEA activities and (2) those. activities, 
thenselveS- 'The third section/will describe the first experiences with 
the modtels of both gnwps. ' . - ' * 

It is 'important in this discHJSSion to recall the purpose of this "field 
t^sf^* and documentation effort. Under Section-151 of ESEA, Title I, the. 
Cotnmiysioner of Education must require SEA's and LEA*s to^.use ''techniques... 
and niethodology-.-.for producing data which are. comparable on a statewide and 
nationwide basis** (Subsection f).^The development and publication of valid 
evaluatiorr models as required also by Section 151 are' the first steps toward 
the nationwide production of cortjparable data; Ut,er necessatly steps are the 
morjitoring of the use of those models and the development Of mean^ to. make 
their "use more error-free. ■ 



Hence, we must re-emphasize that the purpose of this investigation is to 
assess to what degree the evaluation system as now defined (three evaluation 
models, aggregation guidelines, technical docun^ntation, a computer program 
fpr^producing repor^, etc-) can yield the compai*able, uniform data required 
— I5y the legislation. We wil>, also address the benefits States and locals have 
attributed; to the use of the models;. and whenp we find that the use of the 
system has,.jiot gone as intended, vye can pbrsue means to improve the system. 
Hence, the following section desonbes the findings of conversattons with 
vanou? system us^s; 1Jft^.^*inal^ section lists spme suggestions from^theiji 
for refinements!.. \^ ^ . - 

state and local staff c^rimients regarding their expenftences with the models' 

Both State and Toaal distrTct personnel have shown much progress in Cising 
"the models and n6|# ^ready some positive outcomes Vttributable to that 
progress. One thi-ftg especially' welcomed by S|jcite personnel is th^ provision^ 
of comparably data across districts. {Many have never had data reported from 
districts which could be Agdfr^ted or. compared, ) Most States plan to—and 
some already do--use ^he evaluation data to identify especially effective 
strategies within the ^tate- - 
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Another" favorable outcome ha£^beeri the existence gf standards which * > 
States can, use for advising and jnomitoring their J_EAs ' ev^tluation activitiies. 
Some , feel vthat the lack of such information historically ^eft them with^^ 
Httle basis for insisting^upqn specific LEA evaluation practices* ! 

State Rersqjinel havft also notfed the benefits of greater attention tq 
achievement tests—their content, use,, selection, etc. Irr many cases, ' 
testing practices have been^ favoraW v influenced by the models' recommenda- 
tions of certain procedaifes depending upon test characteristics^ That has 
prompteci everyone to look mo^e deepjjy into information about their tests. 

Local, personnel share to a large part the positive reactions to learning 
more about tests* They also welccfcie the possibility of comparing the out- 
comes of thei(^ efforts-to those of districts they know to be similar. In 
maay cases there is also a positive reaction to the explicit statement of 
evaluation steps. *Ttiey feW the recommended procedures are less a burden 
than tneir former attempts meet unclear mandate?. 

Since the system dogs establish standards and recormendations fqr both 
groups, it also enables examination of the extent tp which whose standards 
are being met. In a sense then, on& mu?t acknowledge that any analysis of , 

.evaluation practice^ will yield less glowing comments than were prevalent 
8-10 years ago because at the earlier tim0 there was no way to judge anything 
as unacceptatjle. We solicited the opinions, of our program and evaluation 
contacts regarding their problems, recognizing that the definition of a 
"problems'*' is a relative one.'- We suggested they tfetenpine prob},ems fn. 
comparison to an ideal implementation of the evaluation system, realizing 
that relative to former practices, they often represent considerable 

jmgrovements. ' , ^ ' 

An example, of this soft of dilenma with a problem, that by its very ex- 
istence marks improvement over former practices is the first listed. under 
Data aggregation in Table 3.1. The Title I evaluation system^^empfjasizes 
the aggregation of ?iata only for projects with evaluJitfon data on a\mjor 
portion jof the children served, (projects with minimal attriH;tbn) Tni^ ^is 
to ensurejhat the evaluation data truly reflect the progress of project[ , 
participWts; and before this can even become problematic, one has to have 
progressed from common earlier practices of tabulating pi^etests and postiests 
and never checking that they were for the same children (allowing, it\ the^ 
worst case, the assessment of a groupls posttest compared to a preceding 
"pretest*' of completely different children). Hence, in mani cases, the list 
in Table 3.1 is evidence of an impraj^emfint in local - evaluation methods^. 

Thece remain areas for further improvement, however, and they are discusse 
in three categories: procedural , relating to adhering to suggested rules of 
the models; cl erical , referring to recording, translating', and calculating; 
and analytical , resulting from technical or .statistical problems* 
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Procedural . One area still bleeding work relates to the general 
impleinentation rules. The requirement of Model A that selection of 
Title I participants be based on data other; than from the pretest* 
puffers the most abuse* ^ Even am6ng the States which were the first 
to attempt the models (States with extremely competent Title I personnel), 
the proportion of LEA's still selecting on the pretest exceeded half of 
those, reporting* , 

■ " ' ■ I 

SEA personnel were bes^ with questioris from the LEA's on possible 
methods of fulfilling the model's rules without deviating from their 
present practices or^^perceived /requirements. Various personnel advised 
their fc'EA's of valid w^ys to choose students and still not invalidate the^ 
evaluations, but seldom were the alternative methods tried; or when they 
were.t/'i^d, .they did not eliminate the problem completely. For example* 
one evaluator. provided tables to help teachers in student selection 
suggesting ^choice based on^a combination of statidardized achievement 
test scores, absenteeism, a teacher's estimate of the student's achieve- 
ment, an estimate of underachievement , an estimate of motivation, and 
health. When the reports were' returned , most" school districts were 
found to have used the pretest standardizeij test score'combined with an 
estimate of underachievement (obtained by the subtraction of the student's 
score from the class average). Thus only a single criterion was used and ^, 
. the choice of. individuals to be placed in the Title I group was based solely 
on the pretest. 

The second most (;oninoji lingering procedural problem Is the administra- 
tion oi: nonti-referenced tests at the proper time in Model A* Comparisons 
between the Title I group and the test publisher*s norms are most valid 
whei^'based on real data points. Consequently the tests should, be given^ 
during a four-or six-week period spanning the^est publisher'sjjnain noHiiing 
date. This was seldom the case, although at least the discrepancy between 
true normfng date and date of use is decreasing*.., 

t ■ 

Another administrative problem 1$ that of conmunicating what the new NCE 
metric means. According to a^umb■er of State leaders, parents accept the 
idea with littT^ hesitation, but administrators (especially superintendents) 
resist beyond belief* The use^ of a .vari-ety of metrics for sharing results, 
With interested parties will help ;in this area. 

Administrators also question tge source of^funds for the testing of - 
student? in Title I evaluation, thfe' requirements of Model C include the", 
testing of comparison students froC^mong those not in the project, Jhet*- 
fore if the district already hag btjtajeted funds for" testing once a year, ^ 
and selection of Title I studer)ts^*OTrthat test is acceptable, then Model C 
is a . logical choice. But^'if the money must bg located^ elsewhere, the choice 
can just as logically be Model A, Any modifications of the regulations con- 
cerning who pays for'testing would haye i direct effect on choice of model 
and frequency of testing —perhaps more" of ten; a mpve to Model C wlttj testing 
once a year* (This Involves a small amount of rrfopey usually.) ' 



Clerical . The translation of a raw score into^ny other score is fraught 
with error. In one State, data, from more than 93% of the LEA's contained at 
least one table look-up error. Another State director' reported that the. 
majority of tffrors in his workshop exercises stenin8d,from the ^nappropriate 
use of the same norm table TOtLjboth pretMt and posttest score conversions in 
spite of very obvious table titTfes> Where,yet andther had. modified the format * 
of tha publishers' norms tables,, i^he error Vates were' considerably reduced—but' 
still excessive. Another approach to improving clerical accuracy was the re- 
arrangement of the recording sheet for students' pre-^and post-test scores. 

'Wheththe lEA evaluator converts those scores to NCE's, calculates a . 
different score for each child, and obtains an average difference (gain) score 
for the group, four opportunities fQlt,error exist. (The raw or percentile score 
conversion could be erroneous on the pretest, or the posttest, or the subtraction 
or the calculation of an average could be wrong.' The subtraction of pre^post ^ 
scores, «nd calculation of an average are often double-Checked, but tne score 
conversions are not.) Thus greater imorovement in quality control would likely 
ensue from, a change in the format of tne worksheet as well as that of the con- , , 
version tables. ' . . 

Another problem is that often the gain score foran individual turns out 
negative, and negative numbers appear to be an anathaie to proper calculations. 
One evaluator was so upset by negative scores thafe^he ignored every one in 
averaging gains in his project. f > ' ^' 

The assignment-pf a pretest or posttest score to every individual is only 
part of the process— the two tests must be natched for each individual. In ' 
large districts the matching is generally done on a computer. Here several 
difficulties appear: a matching program must be written; bad coding or punching 
takes a toTl of, properly matched Individuals; a single unmatched card in a file 
makes the entire remalnjng ones mismatched and makes the evaluation worthless 
if left undiscovered; students change the spelling of their names; and in- 
consistencies appear in using tlm"1cr5t jiame first. 



In smaller districts, the match is carried out by hancj. Here the human 
intelligence can remedy many of the problems: -one can see that "Bill Thorn" on ' 
the pretest is "Thorne, W." on the post, for eitample. A knowledgeable - 
individual might remember that a schebl toundary was recently change?!, ahd 
h&yf th^' students are in the neighboring school, or that the Jones children 
oscillate regularly iietween. two schools. Thus the increased difficulty in 
matching by hand is offset by the increased frequency W matches. Sometimes 
the; individual cannot find a match between pretest' andlposttest and simply 
leaves, the space blank. Of the 182 forms seen by USOElstaff, 29 were in- 
complete; the persOTi either overlooked or iinored many litems. 



' A tomparatively minor error in reporting is the failure to include the 
day of, the nwnth on which testing was accomplislied* ^ Under Model A a user 
of the ITBS would be expected to test within a two-week interval on either 
Side of 'Aprt/ 28ttj* If he merely reported that the test was administered in 
App^U the State evaluator cannot assume that the test was given at the proper 
time; it could have been on the first of^April, A similar argument *bolds if ' \ 
May were reported- The error is minor in that It can be corrected easily with 
a change in the reporting requirements, 

Analytical . A Variety of technical' questions dealing with the statistical 
artd psychometric aspects of the system cont1nue\to plague the evaluators* For 
-instance* are they penalizing themselves yihen they test once a year in the 
spring? If students Ypr^et a great deal during the sununer^ perhaps they would 
show-greater gains ^f they were tested fall and spring* When students repeat , 
a grade* what pretest score should be used-^the first pretest or the second? 
And what norms should be used? 

Some State personnel note that the correction of an LEA's evaluation 
error may result in a lowered gain estimate* (They -recognize^ however, 
that no One wants to be fooled iato assuming successes in remediating ^ 
.children*s educational problems if that's not the case*)- ' — - 

The sample of individual project reports perused in the six States 
visited by OE staff were rated on their* quality of evaluation, Those projects 
with the "best" evaluation {as far as we could tell) showed low but positive 
gain scores for the Title I group, (The correlation between evaluation quality 
and size of gain was - ,25) Those projects which appeared to be evaluated correctly 
showed a modest positive impact from th^ program * ' ' 

When the directors artS evaluators ^in the SEA's and the LEA's identified 
the'problems outline*/ above, thy also suggested a variety of solutions* Some 
approaches they had already tried out, and other possibilitieS^rose in dis- 
cussions with USOE s^aff* The suggestions are described in t^next section. 

Suggestions regarding the new evaluation system- 

People we visited and other inter:0Sted parties hav^ ^ggested ways USOE 
can \\^}^> $EA's and LEA*'s follow more completely the pCocedures outlined in the 
evaluation and reporting system* Their suggestions are summarized below. 
,according to the same categories used in preceding sections; procedural, 
clerical > analytical; 
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, Table- 3J Error'rates in evaluation'repoPts 

' jJf.fhe 182,, project reports reviewed,, 32wted no discernabl? error; 

however, the remaiaing^lSO appear to need further improvement in one or , • 

more of the following: . ' ^ 

' ^ , ■ . , Criterion 

I* Test selection/administration > . V Not met 

a. -use apparently appropriate test or subtest, {content-wise) 31' 

b* use test with relevant nonti group ~ ? 

c, pretest at tru^ nonti time 66 

d, posttest at.tr*ue nonti time .58 

e, if sligh^y mi^ss pretest nprm date, make posttest error the 

, same ' 1 

'L—AiS^^s^ level and form pre-and post with Model A ^ , - 14 



g, give make-up tests no more than'2 weeks after original ? 

h/ give lovj enough level of t€;$t to avoid floor effects ^ ' S 

i* .^ive high enough level of test to avoiiTceiiing " * 1 

j* ^gtve appropriate put-of-Tevel test 3 

k., use test ad(nintstrators who are objective ? 

, L follow; standard instructions, timing" ' ^ ^ ^r-^y. ^ 

. - ■ ■ 

aL Scorittg \" u ^ ^ / ^ 

^ ' a. check for^ accuracy of individual student scores , 1 

* 

b. ,inake sure student/score match Is correct ? 

III; Conversion of scores * * .it 

J 

a. use correct conversion tables 11 

V ■ , . ^ . ' - 

^csi-^ do conversions accurately- . * 29 

?. Interpolate between tabled values when necesstfr'y*. '■ , 3 
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^ • ^ Griterion 
IV. Data analysis Not met 

a. use data exclusively from students with'both pre-and post- 
test scores / _ ? 

. b. do all arithmetic accurately . '32 

V. Data aggregation , " ' ^ . 

' a. use data from projects with only minimal attrition 5 

b. check that drop-outs (attrition) are not source of bias ? 

c. make sure studertts not selected for project based on 

pretest scor^ in Model A ' 34 

VL Report form cotnpletlQn ^ / . . 

a. include information about testing dates \ 99 



V 



b. follow instructions * .26 

c; complete all" Items V '29 

d. make writing legible ^ ^7 



Note: The format df the LEA annual evaluation report differs widely from * 
State to State,, rangiit^ fVom the detailed reporting of raW data to supplying 
only a jingle overall measure of Impact. Thus one SEA with access to raw 
data was able to check the data closjply (and found that about 93% Qf the 
reports contained at least one table Jook-up error) ljut another , State had no, 
way to check because raw data resided only with the LEA's., Therefore not 
every, possible problem could be checked by the USOE team and our estimates 
ot various error rates shown in Table 3;1 ar'e undoubtftdli/ low. For^ instance, 
the eleven cases with data basedcDn the use of- irtappropriate nQrms tables (see 
III (a) in the table) are from a subsample of 40 projects jwfiich reported raw 
data^to the State; for those 40, this specific error r^te is 28%i we do not 
I^MHor the other 140 or so reports examined. ' / 

^ A ■ 
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Pr(teedura1 : Most of these apply to the generai.implementatioft rules 
and Other administrative, areas* * < 

^ TT"^Pm44e_a^et^ special. handtiDOk OfTthe implementation of each 
moctel* Th6 haridBooIr ^ouW be very eTementary, in step-by-step f]ow chart 
f&sntpn, with plenty of concrete examples of documents ^nd approaches which 
have worked* . , ' 

^^.^ 2* Emphasiw reduced testing requirements with the proposed models* Too 
many school personnel are too worried about t^oo mdny tests'* Specify, for each 
model, the minimum testing possible*. - ' 

3* To encourage the proper administration of test^, encourage tho^e 
districts using onte-a-year spring testing to have the teacher from the next 
.highef grade give the test. To the third grade teacher testing the second 
'"^grade students at the end of the year*, accuracy woujd be paramount since next ^ 
Jyear he i^uld have those very students and would, supposedly, welcome accurate 
test scores in tjieir folders* 

4- Give more guidance regarding test selection. Many studies have 
demonstrated the importance of test content for detecting student growth in 
specific skill areas, 

5- Add more infOrmatiorn to the handbook on out-9f-level teeing* Host 
individuals still feel very uncomfortable attempting to implenent functional- 
level testing although they recognize the necessity with some students* Pre-, 
pare a detailed checklist which Jtates can give to LEA's showing the effort 
involved in ordering^ the logistics of ..testing, and the scoring and translating 
of sc;ores** Discuss the pay-offs for these e'xtra steps in terms of an increase 

" ,in accfjracy of the scores. Also, som^ n^yths are common and should be Investigated 
.and discussed* ("Functional-le^^l testing will result in th^ pretest average; t 
being lower;" "it will resutt m ^choosing the wrong students;" "it will give/^ 
"-inaccurate estimates of gain;" "it will compare students at one grade unfairly 
with those at the next lower grade, etc* ) j J / - 

6* Provide^guidelines on what action to tsike if the raw score, leads/tp a 
converted score too low to be incited in the percentile table* For exanjple, 
a fourth grader might take^ji out-6f-level Metropolitan Achievement TesV>\ 
Totals Reading* The raw score couYd correspond to a standardized score .6^ 30, 
, but ttt^t'is top lo^K to 4)e included in the percentile table* - / ; 

" , ' ^ ■ ' . § ,\ ^ , 

'"7- eorrtmunicate the results of the data collected in thjs study trtf publishers, 
'especially the information about the needs for norming^earller in the jfear,' and 
for less confusing norms tables. . ^ ^ ' - ^ ;/ 

8* 'Renove the^suggestipfl in current documentatiorv that two-thfrds^ of the 
project should take'plade between the, two pre-and post-tests* The fr^idence of 
, failure to follow this, requirement is negligible, and shou,ld diminf$h,Jto zero 
as (fistpicts fiiC^ve to appropriate testing dates* The requirement Vea<fs/ to 
refilling of non-informative data, and to prob^lems when schools have ^ |irov1sion 
for students to return^ to their, regular classrooms >/hen they have mastered a 
certain body of material. ' {Only Z.Z% of the projects sampled did ftot^meet this 
requirement, and they allplanned to remedy the problem in the next school year*) 
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9* Investigate the conditions under which combining results frati 
differerft grades is appropriate* If a' comparison group for Model C is too 
small within a particular grade, some addition of non-Title I students from 
the next higher grade might be possible although technically they are not ift 
the project* Or in Model A, adding the two Title I students in 8th grade to 
those in 7th might reduce the trauma attenoant to findir>g an average of a 10 
NCE Toss (due to small sample size and u|rstable data)*" ^ . , 

10* In thelight of the high rat6 of errors in table reading, cpnslder 
the use of i raw score reporting system* In the absence of mecbatiical aids 
to table lopk'Up, provide simpler scorer conversion tables to users, aod 
communicate the problem to test publishers* (Some publishers' have already 
provided vastly improved tables for those States wheret the evaluator has 
insisted upon them.) ' * ^ \ . * 

11. Provide assistance, both in guidelines and "in the computer program, 
for the plotting of pretfiSJLagainst posttest for all models. Tlje visual 
inspection of the. scatterplots could be expected to indicate fToor an/d 
>ceiling effects, gross non-|inearity of relationships, and the presence ^f , 
unexpectedly high or low "gainers.*' 



12* Classify evaluation errors by severity so that when two Mmplejnenta-* 
tion.suggestions conflict, the, Title I person can have some guidance'. 

*^ 

13. Many pf the details of thg'proposed system are best communicated 
in materials to supplement a^ workshop. The technical reports- are excellenW^ 
but difficult for SEA's and LEA's to obtain (budget problems, communication 
concermng new reports and methods of obtaining them* etc)* Make^hem" 
available "to &EA"s at no cost; SEA personnel could distribute them giving 
implicit sanction and perserving their leadership* 

1 4. Sh6w some examples ^f how program objectives cooTd be stated under 
the new reporting format* " ' ■ - 

One State requests that the LEA Title I ''director estimate the NCE gain t(>* 
be achieved for each grade within each project. If the third g«:ade l^eading ■ 
project at Memorial Elementary has a reputation as the best, tjben a seven 
NCE gain may be the objective. If it H the worst, then one-balf an NCE 
might be appropriate. - v 

]5* Prepare examples to help SEA'S and LEA's communicate evaluation 
results to their various audiences, *including"use of NCE's, perce>itile$> etc. 

. . ■ ' " * 

16* Consider alternative methods of tying a cost. figure to a pf^olehj* 
Since districts generally spend about 75% of their budget^rjnstructionar 
personnel direct costs, fepor^^ing only those costs iflay ^flWBse the accuracy 
■ and decrease the reporting burden. Similarly,^ there may b6 a way to rely 
merely on GgPA Section 437 data. ^ , 
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. ! Clerical. -CleHcal suggestioVis refer to recording, "translating, and 
calculating processes., .1 

1. Instruct I'EA esraluators, vifien looking, up average standardized scores- 
in pu^Jlfshers'' norms tables, to usel the, individual percentile norms table, not 
the school norms tabre/ ' \ ^* . / - - 

2; Remote any/reqijiranerits for^ dat^ point Interpolation at the district 
leveK Inter^gl^tion appears tc^ be ipope' ef*ror-prone than i<^is worth'. 

J* , 

3. Send copies of an exemplary testitig-dates chart to all- interested . 
Mrties. For example, the forms used in one State include a chart such as "the 
one shotm in Appendix, 3' which assis,ts district personnel in avoidtng test 
administr^ ion d ^te er rors . ' * ^ 



* 



. 4. Add» requireaent for>the project re^rt to provide the average seTec- 
tioit'flCE, whjgj-g possible, and the average pretest NCE. This will allow an easy 
edit check to s^e if 'the selection was based comtiletely on the pretest, and 
^iL^il^ wst' fibedjf students" were chosen. * 

5. Revise the percentile-to-NCE conversion tables so that they .will be . 
ea'sier to use. Perhaps they could be placed in groups of ten, with clear Ifnes, 
to demarcate the columns. - "^hey are error-producing at present.. , 

6/ Develop "opt1*scan" forms and software for the analysis of scores from 
major tests. If a State, or large LEA, decides to centralize the scoring pro- 
cess, rather than have the local classroom teachers, score their own papers, 
this' is the solution with the greatest long-range potential, though perhaps 
the highest initial expense. Eurthermore , though the accuracy and speed of 
daU processing will be vastly improved, ^the systap myst also have* a set of 
appropriate error-detection methods built in* * 

7. Encourage lEA's to icore some tests by band, even if they have engaged 
a scoririj service, in order tocKeck:.,the accuracy the sem'ce. 
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8, Provide guidance regardfng. tftfe. use.of automated data processing as 
often as possiblefor the various scpre conversions and manipulations 
necessary in the system, .Encourage oth^ approaches, too, to preserve the 
integrity of the data; 

(a) Staff should try -t(i perform scor^ conversion activities in teams, 
with people double-checkilia^the work of oihers whenever possible; 




(b) Raw data should be stored (or senljt to the LEA or, SEA) to enable 
later chewing of a s^tmple of them regardi^ig the correctness of the tab'les 
used, tire score coPl?ersions> etc*; 



^c) Tables/should be re-formatfed at 
easier readincr (e*g, , 2-col(3mn tables are 
multiple-coliimn' tables); and, 



qhe local level to allow for 
Jch easier to work with than. 



(d) 5taff responsible for performing the.so6i»e conversions should be 
trained }n the use and interpretation q.f standardized tests, so that the 
scores ^at they work with will c6me'to havei intrinsic meaning leading to 
detection of clearly wrong, inappropriate* or but-of-range scores. 

Some sort of pre-programnfed ^^3tay,?^or^clr igimputer can also facilitate 
the data analysis. For small LEA's or^EAs wHHSuJt access t(> larger computer^ 
facilities, a set of programs and implementatioq^materiali are being developep^ 
for use with hand-held programmable calc;ula1;pns Available from $89 ^intl up* / ' 
(Resource Development Institute, the J^fin^fesll ^ Center serving A 

Region VI, has developed" these program^^ wMdhVcan be furnis-hed on magn6ti^[ 
cards .for at least two machines, the^JejcaS IiS^ruments Programmable 59 and . 
. the Hewlett-Packard 65.) ' ' \ ^ 

Analytical . The suggestions In tft^.&lctil 
characteristics of the models* -^"^ 



)n refer to tedinix;al 



1- Clar'ify the severity of the regression hazard when two-stage selection 
takes place. For instance, a potential'Title I treatment group o*f 100 students 
may be iderjtified by teacher" referral* Then a pnetest is given to those 100> 
and, the 95 students most in need are given treatment* Obviously the' reg>*ess,ion 
effect is considerably less than if the f>ve studifnts most in need were chosen* 



ring project effects from/ 
ir testing is mtrch easier on 
le loss of subjects and their 



2. Investigate further the'trade-off in com) 
spring to spring versus fall to spring, Once-a-yel 
the budget and school time> but may be offset by tl 
loss of knowledge over the summer/ ^ 

3. Assist evaluators in the objective identification of "outliers," 
(stu(Jent dat5i sq extreme that th^ are likely ip ernor)* For instance*, in 
plotting a Model C implementation* o^^istrict evalyator was surprised to- 
find two comparison group individuaMBfo were at thi top *of the. distribution 
on the pretest and almost at the bMxoln on the posttast* - Examination of their 
scores revea^led that they had, scored at the 99th percentile on the reading 
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comj^rehensxpn pn^^^^ithout missing a single item. In contract* their 
vocabulary sfeores w^re at the 1st percentile. Clearly some errot:\had been 
made (by'ttie scorir^. ser3/tjee> perhaps)* a.nd the individuals were dipped from 
the analysis. The result was that the estimate of project effects .changed' 
from negWiv^to positive; ' ' ^ ^ 



Three conclusions appear evident from Chapter III: (1) The process to 
change LEA evaluation activities to conform with tdose prescribed by the 
evaluation models is^aborious and* by necessity* iterative- States we 
visited had staff pursuing this goal for as long as two years* and many re* 
ported that more work is still needed, * 

^C2|^..RcoijftMy the^jTjQst pervasive. administrative problem for SEA's is that 

it is . almost impo^sibl e for tKem to know what actually happens in the LEA*s. 
Of course* this is much more than just an evaluation [Jrablem* but it greatly 
affects evaluation data. The tests may have been given oh other than the 
reported ^ates (in one of the States* the test was supposedly administered on ' 
a Sunday); tTie test administrator may have ignored the directions fy proper 
test giving; tests designed for group administration may have been given *^ 

Nndividu^ly or withrelaxed time limits; they may have bei^n administered to 
the wrohg individuals; or other situations may have prevailedSnd no one will 
ever know. ' ' ^ ■ 

(3) Another threat to the validity of data reported appears. to be the 
reliance on ^^^nformation that is scored* transformed and analyzed manually. ^ 
Though conversion to automatic data processing wherever possiU16,in the 
evaluation system does not promise to be a total panacea* it seems a promis- 
ing first step. / ; * 
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^GHAPT€R IV. REGULATIONS 



TMs report' .ha%',discu^'?d the ^^tperiences of SEA and LEA personnel who^ 
used t.he evaluation models duri'np spEiff^tyear 1576-77_ (and in a few cases, " 
earlier). All State stsff f^ve peerf encouraged to get the LEA adojxtion 
process underway* havB. attended na,t1biiaj Vorkshpps regarding the models* and 
have called on thejp Technical Assistance Center consultants for help; USOE's 
role 4,n this has been otie of encouragement* p'erslrasivjgRess (usually,), and 
assistance. He anti'&iijate that those behaiiors.will^xon'tinue qven as use of- 
,the uniform evaluati^fi Inodels becomes reqt^red, not merely voluntary. ^The^ 
^ purpose of this chapiter is to discuss our work- underway to Issue such a re- 
quiranent in the forTn of> regulations. 

_ In respjonse to' Corlgressfpnal inquiry fn early 197-5* we sough^a 1^9al ^ '. 
opinion regarding the need* for regulations and were advised that they 'are 
required fcr Section IBI. The following paragraphs describe (t) our efforts 
to solicit fnput from interest gr^ps and the public, (2).the probable content 
of regulations* and (3) the anticipated publica>tion procedure and schedule. 

Public Participation .and ConsuTi^ioni . As indicated in pur earlier .reports 
about the model developntent process we |b|y%..solicited the advice and Sbqges- 
tions of SEA and LEA representatives rOTat;ding appropViate models for Trtle I 
evaluation. We wtTl coMiJiue our effotttsto consult with the public durrfngj 



the, regulations proems. \ - * ' \^ 

, In the fall of, 19^» in- anticipatfon^ot the preparation 'of Pegt(latJons, 
aperies of informal mel^tlngs #fas hel4.A;ith, representatives *f the Cfeffimittee / 
on Evaluation and Infontiation Systems (MIS) bf the .Council of Chief State 
School Officers a^d wjtK States, J^tle^J *CdO)^inators. ' In addition* we have 
been contacting .r^r'esentcit,iv/ss* of each oi\p;he major interej^t groups in 
education to inform them.o^ Pur Vork io.'^Ie, and to advise them that re- 
/ gulations are fortheomin^.;^, the.re^uT^'J^^ process will al-so include an 
opportunity for public c^i&tr^ after-botti a memorandum ^ut our intent 
and Proposed Regulationslire pistribUted.' , ' 

Regulation Content .r"$6ctipn.l51{6) requires' the Coirthisstoner to publish , . 
"standards of program or 'prp3^ct*effectiye(jess*'' In keeping, with this require- 
nfentr'the regulations wiiri,iTtl,ade^,a set of general teahhicjl standards 
' approffriate for TItU I eya^l.aatl'bijs.^ Thes^ standards will be concerned with 
such issues as. the use^ Mnmarispn group's, the conditions under which sampl- 
ing i§ appropriate; the rellaoflity and yal.idity of tests," etc. Th^estaliTish^^? 
ment of general standiirds is soiftewhat' problematic because their apprapriate 
application to a spfec,ific TitJ^ I -project may. vary depending on the 

fr> 'Characteristics of that proS^i>^ In trying'to develop the best formulation 
bf techni<:al standards* we afe^Jc'pnsulting the wirk of the' Joint Committee on" 
Standards for Educaifopal E)4l«atioiv{a joint committee representing^the major 
. professional organizations i»-, education), and are also doing further''empir'ft;afl 
wprk^to detertnin6tth.e dtegree^bf spetification that the standards must contain 
in order to ensure that^the rfata obtained will be 'technically sound. This 

, J., 'work is concerned, with such questions as how reUabte ao instr;Ument must be ^ 
in order for rfn evafluator to have confidenceVin t^»e data ityields* etc. 



Tf)e three models, that are currehUy being disseoilnated describe ' ^ 

pKOcedures which, when correctly followed, will yield data that are 
technically 4ound* The, regulations will describe the major requirements 
of these three models* /We anticipate that raostLEA'swho are providing 
instructional services in reading and math in Grades 2 through 12 will use 
one of the three fliodelsV However, art L.EA (or ah SEA) might wisji to develop 
an alternative to. then)* Stich alternatives would.be permitted, provided that 
tbey aire techhicaTly sound and meet the requirements of the legislation* To , , 
accdmmodate the devSlapment of acceptable alternatives, the regulations will 
specify a process, for their approval* We should note that these comments ^ " ^ 
reflect our current thinking about regulations; that thinking may certaia\y.^^ r 
beLitodifie^^As a consequence of suggestions received from the public durin^^ ,V 
the regulat:ior>s.^ process* . ^ ' ^ * 

' ^ ' ^ 1^1 ^ ■ 
. ^-^ ^.^.^ 

> The Regulations Pro'fcess and Schedule * Our Reguljitions'process will . * ^ 
involve t1) the distribution of a memorandum- to, all States which. solicits 
colmi^nts about specific issues regarding upcoiifiig regulatioos; (2) a. y5iijCe..Qf^ . _ 
Proposed llule-making, and (3) Final Regylatiphp* Delays in the many 
clearance steps required for thfe^ publication of regul?^tions have forced 
us tof^ revise our publication schedule, and earlier plans for a formal 
Notice of Intent wer^ not^proved-- by the Office of thtf Secretary* , , ^ 

> '.^ , 

He hope to publish Proposed' Rules in the^flU of 1978, and 
Final Regulations duririgithe foUowingj^inter or spring* This means that 
regulations governing Section 151 would be in e^ejjti>during .the school year 
1979-80* However* as indicated in Chapter III,,, (feny States will g.o through ' 
several steps in implenienting the new >egulatibn$>*and there will be con- 
^siderable variation in the speed with which different States achieve 
'implementation. Those who already trying the models on a^volunl^ary 
basis;shoUld be abTe to report data from the school year 1979-80 in tiieif 
November 1980 report to the Commissioner* Other States will require a 
longer impleJnentation period* Thisoneans^that USQE's receipt of unifprm data 
from local and state reporting will probably increase gradually through the 
next few years. We have work underway now— which we will report on by the 
end of the' summer, 1978—to estimate the amount of aggregatable dstta to be 
received each year and the degree to .which national estimates 4nay result 
from them {with assorted accompanying figures to estimate, the effects of 
missing data, etc,)* - r . 
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, , ' ■ ' CHAPTER vT CONCLUSIONS 

The three -SubstSintive chapters of this report described (a) th^ 
current status of the use of new Title f evaluation models nationwi* 
(b) the experiences of State and local personnel who have tried the >^ 
procedures and their reactions and'(c) the process underway simultaneously 
in USOE to draft and publish regulations in this, area. This documentation 
/tfT the evaluation program in terms of various sctivitt^s and some of their 
' " effects also suggests sortie further work to facilitate and SEA evaluations 
Df Title, L 

The purpose of this chapter is^o highlight some ojft^r conclusions abo6t 
^the work at this time and to desct^be briefly some of tm activities planned 
for the future which are implied by'' those conclusions* The chapter is 
organized to address four major top^ics: {1) services to support the use of 
Title I evaluation models, (2) documents tg further describe .the models, (3) 
fdcused^ inyestigltTohs on^^en^ and (4) sol idi tatiOn' apd use 

of public input ^ ^' 

Conclusions and pla'ns regarding assistance service 

• Discussions with SEA and LEA personnel have continually emphasized the 
irrtportance of on-site, specific advice regarding evaluation activities. Often 
principles c&n be stated and general advice offered in handbooks, training 
sessions, etc., but unique characteristics of projects, particlpafii?, and/or 
setting demand that special consideration come into play for the application 
of those principles- Personnel in.the Technical Assistance Centers play a 
vital role in this area» and current indications are that they fulfill the 
role *e3££remely well- ' " - 

Now that the Centers have been ifl operation for nearly two y&ars, we feel this 
is a/i appropriate time for a third-party assessment of their work and function, 
A contract to support such an effort is underway as well as continuing discussions 

with State, and local personnel. 

' ■ , - t ■ 

SEA'S have^een surveyed to determine their anticipated need for services 

during the period enduig^March 1 , 1979» and the Centers have been funded to meet 

that need. {There are also provisions for meeting increased neejds if estimates 

prove to be too low)- Federal regulat>on5 dittate that we conduit a competitive 

■ procujemen^t to award the^ntracts for provision of these services after 

March of 1979- Our Intention is to Insure, however, that services not be 

interrupted during such a procurement. Other services, as provided by USOE 

personnel directly of^ as augmented by additional contracts (for example, f-or 

the prov1"Sion of^tgrial-s and assistance with national workshops to introduce 

SEA personnel to e1?ajruation models^jfor the Title 1 ^proqr^m for^the * 

^ neglected or delinquent) vri 11 also'continue- y ' ' * - 

Docunifints about the moc^lj; and their procedures 

' ^ 7 

USOE policy ha^^^en that the, Gov erf¥nent Printing Office would be the 

primary agent for distributing handbooks, pamphlets».and o^her materials abdut 

the proper conduct Title I evaluations. It>was feltvtfiat printing and ^ 

dist^^tution of perrr^q^s 20,000 copies of documents would be impossible for 

^^V^ipE ^ acc6mpl-ish. 



Difficulties faCed by 3EA's and LEA's in obtaining our materials have 
made us realize that our early decision^(to rely primarily on GPO and only 
supplahent those efforts]) Ws wrong. We now envision a more active role, for' 
USOE.in the prbvistor^ of materials—contacting States to determine the number 
of copies needed, having a sufficient number printed, and mailiJig them from USOE, 
We wilT^^lso mainl^aa stock of USOE materials in thr^echnicai Assistance 
Centers for their W'sonnel to distribute. The third source of materials will 
be GPQ to enable pPsons who. do not interact wrth States or the tenters to . 
obtain copies. ^ . : ^1 

Another effort growing froift^SEA and L^r*equests will be computer soft- 
ware" and documentation which is more generalizable than our current version. 
It will include proVsions for entering data in various formajts, for performing 
edit check? on the data^ for printing out scatter-plots , etc* We hope this will 

, improve the accuracy in computation, etc.*--tbat,is JiOVLiujDJ^bl^ra^—Hi^ 
will begin under contract in fiscal year* 78, and copies of t^e pwgram and 
documentation will be distributed widely as described above. 

♦ * * 

^ Persistent confusion regarding specific decisions- in the conduct of an 
evaluation as well as recent recomfnendations by the Lawyer's Committee for 
Civil Rights under Law under contract to NIE (Silverstein and Schember, 1977) 
have identified the need fOr a technical policy manual in the area of Title I ^ 
Valuations. Our plans are that it will be organized topically to facilitate 
referencing it and will include many examples. It will provide guidance abou.t 
specific practices Qjrf^ implied generally by tt(e regulations, and we hoPe to 
have a draft availaole for review by interested "pa^rties. simultaneously with 
their review of draft rules., 

' Focused investlqationji to answer remaining questions 

Problems still arise in districts regarding decisions about specific ev^lua- 
**^tion practices. Many of those decisions involve trade-offs about which ther 
is currently no information^' For example, some investigators have note4, in- 
^accuracies in scales across test levels developed to allow the use of a. lower 
or "functional level" test with students and still describe their performance 
with re*spect to that of their age-level peers. Others note that there are 
inaccuracies, too, when students ar^^given tests too difficult for their de- 
velopmental level. Local evaluatorsC then, must judge which typ^of error 
will be worse fthat due to the scaling prol^ems or that due to a mismatch 
between items and/^student level), a;id they need more information by which to 
^make such judgments. We plan to re-analyze existing data bases (from completed; 
Federal evaluation studies) for eVidence regarding such trade-offs, and that 
evidence will be^much oC^the content of the technical policy manual* described 
above. 

^ Work is also underway Vegajtding evaluation models for other parts of the 
^ Title I program, such as tha\ providing services for migrant children, for the 
neglected pV delinquent, and fdr pre-schoolers. Each is being addressed iji a 
separate contract, and USOE, Srate, and LEA personnel serve on advisory pane^ 
to react to feasibility ^nd validity issues as the work proc^^ds. 
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A third area of current inv.estigation is focusing on the amounts of 
usable LEA and SEA data USOE will have at different times for reporting to 
Congress. We plan ta estimate these amounts and develop means for using 
them by next fall* (AH that time we wiTl also have more definite informa- 
tion about the timing of regulations* allowing ^s. also to predict when we 
will have uniform>*coin^arable d^ta from all SEA' sO 

The importance of continued public input ^'j 

* 

Section 151 emphasizes SEA am^_EA involvement in this work to develop, 
evaluation models, and the progt^am to implement the Section has relied heavily 
on their input* SEA and LE^^rsonnel are on panel^^focusing on model develop- 
ment activities for e$ch of the parts of Title I; aSditional conmljttees fjave " 
been formed to assi st wijbnfcjie drafting of regulations; : ^ ^ 

There is/tfujch interacfTOn-^among personnel f^om USOE (both in Title I 
and In evaliiation)*SEA's (again representing both the pwgram Snd evaluation 
areas)* L^'s, professional groups* and conmercial concerns (for examplevtest 
publishers)*' Seldom is there unanimity across these groups regarding policy 
for Tj^tle I evaluations, but the participation of people with varying concerns 
-can at least keep USOE recommendations from being more restrictive than necessary 

In short* it is rewarding to see changes which mark improvement ^ 
the state-of-the art of educational evaluation* One ca;i cite* for example* 
cljanges in the types of scores and services. offered by test publishers; the 
increase in the number of textbooks ort evaluation topics (often including 
the s^me topics as those emphasized in the;Iitle I materials); more 
discussion at professional meetings; critical examination and use of i 
evaluation data for identification of effective practices or for changing" 
programs; and increased^interest on the P^^rt of parents* While it is not 
appropr4atje to assume that the Title revaluation system has been solely 
responsible for this trend* we do^feftJ it has played an important role in 
it and has been* in certain instances>.a catalyst both for the'improvanent 
of evaluation practices and for the furwer examination of related issues* 



